Papers with large multimodal model
FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model (2024.acl-long)
Copied to clipboard
| Challenge: | Existing image captioning evaluation metrics do not provide an explanation for the assigned numerical score. |
| Approach: | They propose an explainable reference-free metric to provide an explanation for captions . they introduce score smoothing to align as closely as possible with human judgment . |
| Outcome: | The proposed metric achieves high correlations with human judgment across image captioning evaluation benchmarks and is publicly available at https://github.com/Yebin46/FLEUR. |
Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-aware text-based visual question answering methods are based on textual entities in images. |
| Approach: | They propose a visual text entity linking module that harnesses a state-of-the-art visual text recognition engine and the power of a large multimodal model to perform visual text-entity linking. |
| Outcome: | The proposed approach surpasses the previous best approach by 23.3% on an absolute scale and establishes a new state of the art. |
Aria-UI: Visual Grounding for GUI Instructions (2025.findings-acl)
Copied to clipboard
| Challenge: | Using a multimodal model, GUI agents can ground from language instructions to target elements . relying on HTML or AXTree inputs is a challenge for GUI agents . |
| Approach: | They propose a large multimodal model specifically designed for GUI grounding that adopts a pure vision approach instead of auxiliary inputs. |
| Outcome: | The proposed model outperforms vision-only and AXTree-reliant models on offline and online agents. |